

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>CephFS Snapshot Mirroring &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/underscore.js"></script>
        <script src="../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="客户端配置" href="../client-config-ref/" />
    <link rel="prev" title="Snapshot Scheduling Module" href="../snap-schedule/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../">Ceph 文件系统</a> &raquo;</li>
        
      <li>CephFS Snapshot Mirroring</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../_sources/cephfs/cephfs-mirroring.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../">
          

          
            
            <img src="../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../">Ceph 文件系统</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../#cephfs">CephFS 入门</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../#id4">管理</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../createfs/"> 创建 CephFS 文件系统</a></li>
<li class="toctree-l3"><a class="reference internal" href="../administration/"> 管理命令</a></li>
<li class="toctree-l3"><a class="reference internal" href="../multifs/"> 创建多个文件系统</a></li>
<li class="toctree-l3"><a class="reference internal" href="../add-remove-mds/"> 配备、增加、删除 MDS</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/">术语</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/#mds">MDS 守护进程的引用</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/#id3">故障切换的管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/#mds-standby-replay">热备的配置</a></li>
<li class="toctree-l3"><a class="reference internal" href="../standby/#mds-join-fs">配置 MDS 与文件系统的亲和性</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cache-configuration/"> MDS 缓存配置</a></li>
<li class="toctree-l3"><a class="reference internal" href="../mds-config-ref/"> MDS 配置选项</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../man/8/ceph-mds/"> ceph-mds 手册页</a></li>
<li class="toctree-l3"><a class="reference internal" href="../nfs/"> 通过 NFS 导出</a></li>
<li class="toctree-l3"><a class="reference internal" href="../app-best-practices/"> 应用最佳实践</a></li>
<li class="toctree-l3"><a class="reference internal" href="../fs-volumes/"> FS 卷和子卷</a></li>
<li class="toctree-l3"><a class="reference internal" href="../quota/"> CephFS 配额管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../health-messages/"> 健康消息</a></li>
<li class="toctree-l3"><a class="reference internal" href="../upgrading/">升级 MDS 集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../upgrading/#firefly-jewel">升级比 Firefly 老的文件系统，需过 Jewel 这个槛</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cephfs-top/"> CephFS Top 工具</a></li>
<li class="toctree-l3"><a class="reference internal" href="../snap-schedule/"> 定时快照</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#"> CephFS 快照镜像</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#requirements">Requirements</a></li>
<li class="toctree-l4"><a class="reference internal" href="#creating-users">Creating Users</a></li>
<li class="toctree-l4"><a class="reference internal" href="#starting-mirror-daemon">Starting Mirror Daemon</a></li>
<li class="toctree-l4"><a class="reference internal" href="#interface">Interface</a></li>
<li class="toctree-l4"><a class="reference internal" href="#mirroring-module">Mirroring Module</a></li>
<li class="toctree-l4"><a class="reference internal" href="#bootstrap-peers">Bootstrap Peers</a></li>
<li class="toctree-l4"><a class="reference internal" href="#mirroring-status">Mirroring Status</a></li>
<li class="toctree-l4"><a class="reference internal" href="#configuration-options">Configuration Options</a></li>
<li class="toctree-l4"><a class="reference internal" href="#re-adding-peers">Re-adding Peers</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../#id5">挂载 CephFS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../#id6">CephFS 内幕</a></li>
<li class="toctree-l2"><a class="reference internal" href="../#id7">故障排除和灾难恢复</a></li>
<li class="toctree-l2"><a class="reference internal" href="../#id9">更多细节</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="cephfs-snapshot-mirroring">
<span id="cephfs-mirroring"></span><h1>CephFS Snapshot Mirroring<a class="headerlink" href="#cephfs-snapshot-mirroring" title="Permalink to this headline">¶</a></h1>
<p>CephFS supports asynchronous replication of snapshots to a remote CephFS file system via
<cite>cephfs-mirror</cite> tool. Snapshots are synchronized by mirroring snapshot data followed by
creating a snapshot with the same name (for a given directory on the remote file system) as
the snapshot being synchronized.</p>
<div class="section" id="requirements">
<h2>Requirements<a class="headerlink" href="#requirements" title="Permalink to this headline">¶</a></h2>
<p>The primary (local) and secondary (remote) Ceph clusters version should be Pacific or later.</p>
</div>
<div class="section" id="creating-users">
<h2>Creating Users<a class="headerlink" href="#creating-users" title="Permalink to this headline">¶</a></h2>
<p>Start by creating a user (on the primary/local cluster) for the mirror daemon. This user
requires write capability on the metadata pool to create RADOS objects (index objects)
for watch/notify operation and read capability on the data pool(s):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph auth get-or-create client.mirror mon &#39;profile cephfs-mirror&#39; mds &#39;allow r&#39; osd &#39;allow rw tag cephfs metadata=*, allow r tag cephfs data=*&#39; mgr &#39;allow r&#39;
</pre></div>
</div>
<p>Create a user for each file system peer (on the secondary/remote cluster). This user needs
to have full capabilities on the MDS (to take snapshots) and the OSDs:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs authorize &lt;fs_name&gt; client.mirror_remote / rwps
</pre></div>
</div>
<p>This user should be used (as part of peer specification) when adding a peer.</p>
</div>
<div class="section" id="starting-mirror-daemon">
<h2>Starting Mirror Daemon<a class="headerlink" href="#starting-mirror-daemon" title="Permalink to this headline">¶</a></h2>
<p>Mirror daemon should be spawned using <cite>systemctl(1)</cite> unit files:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ systemctl enable cephfs-mirror@mirror
$ systemctl start cephfs-mirror@mirror
</pre></div>
</div>
<p><cite>cephfs-mirror</cite> daemon can be run in foreground using:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cephfs-mirror --id mirror --cluster site-a -f
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>User used here is <cite>mirror</cite> created in the <cite>Creating Users</cite> section.</p>
</div>
</div>
<div class="section" id="interface">
<h2>Interface<a class="headerlink" href="#interface" title="Permalink to this headline">¶</a></h2>
<p><cite>Mirroring</cite> module (manager plugin) provides interfaces for managing directory snapshot
mirroring. Manager interfaces are (mostly) wrappers around monitor commands for managing
file system mirroring and is the recommended control interface.</p>
</div>
<div class="section" id="mirroring-module">
<h2>Mirroring Module<a class="headerlink" href="#mirroring-module" title="Permalink to this headline">¶</a></h2>
<p>The mirroring module is responsible for assigning directories to mirror daemons for
synchronization. Multiple mirror daemons can be spawned to achieve concurrency in
directory snapshot synchronization. When mirror daemons are spawned (or terminated)
, the mirroring module discovers the modified set of mirror daemons and rebalances
the directory assignment amongst the new set thus providing high-availability.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Multiple mirror daemons is currently untested. Only a single mirror daemon
is recommended.</p>
</div>
<p>Mirroring module is disabled by default. To enable mirroring use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph mgr module enable mirroring
</pre></div>
</div>
<p>Mirroring module provides a family of commands to control mirroring of directory
snapshots. To add or remove directories, mirroring needs to be enabled for a given
file system. To enable mirroring use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror enable &lt;fs_name&gt;
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Mirroring module commands use <cite>fs snapshot mirror</cite> prefix as compared to
the monitor commands which <cite>fs mirror</cite> prefix. Make sure to use module
commands.</p>
</div>
<p>To disable mirroring, use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror disable &lt;fs_name&gt;
</pre></div>
</div>
<p>Once mirroring is enabled, add a peer to which directory snapshots are to be mirrored.
Peers follow <cite>&lt;client&gt;&#64;&lt;cluster&gt;</cite> specification and get assigned a unique-id (UUID)
when added. See <cite>Creating Users</cite> section on how to create Ceph users for mirroring.</p>
<p>To add a peer use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror peer_add &lt;fs_name&gt; &lt;remote_cluster_spec&gt; [&lt;remote_fs_name&gt;] [&lt;remote_mon_host&gt;] [&lt;cephx_key&gt;]
</pre></div>
</div>
<p><cite>&lt;remote_fs_name&gt;</cite> is optional, and defaults to <cite>&lt;fs_name&gt;</cite> (on the remote cluster).</p>
<p>This requires the remote cluster ceph configuration and user keyring to be available in
the primary cluster. See <cite>Bootstrap Peers</cite> section to avoid this. <cite>peer_add</cite> additionally
supports passing the remote cluster monitor address and the user key. However, bootstrapping
a peer is the recommended way to add a peer.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Only a single peer is supported right now.</p>
</div>
<p>To remove a peer use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror peer_remove &lt;fs_name&gt; &lt;peer_uuid&gt;
</pre></div>
</div>
<p>To list file system mirror peers use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror peer_list &lt;fs_name&gt;
</pre></div>
</div>
<p>To configure a directory for mirroring, use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror add &lt;fs_name&gt; &lt;path&gt;
</pre></div>
</div>
<p>To stop a mirroring directory snapshots use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror remove &lt;fs_name&gt; &lt;path&gt;
</pre></div>
</div>
<p>Only absolute directory paths are allowed. Also, paths are normalized by the mirroring
module, therfore, <cite>/a/b/../b</cite> is equivalent to <cite>/a/b</cite>.</p>
<blockquote>
<div><p>$ mkdir -p /d0/d1/d2
$ ceph fs snapshot mirror add cephfs /d0/d1/d2
{}
$ ceph fs snapshot mirror add cephfs /d0/d1/../d1/d2
Error EEXIST: directory /d0/d1/d2 is already tracked</p>
</div></blockquote>
<p>Once a directory is added for mirroring, its subdirectory or ancestor directories are
disallowed to be added for mirorring:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror add cephfs /d0/d1
Error EINVAL: /d0/d1 is a ancestor of tracked path /d0/d1/d2
$ ceph fs snapshot mirror add cephfs /d0/d1/d2/d3
Error EINVAL: /d0/d1/d2/d3 is a subtree of tracked path /d0/d1/d2
</pre></div>
</div>
<p>Commands to check directory mapping (to mirror daemons) and directory distribution are
detailed in <cite>Mirroring Status</cite> section.</p>
</div>
<div class="section" id="bootstrap-peers">
<h2>Bootstrap Peers<a class="headerlink" href="#bootstrap-peers" title="Permalink to this headline">¶</a></h2>
<p>Adding a peer (via <cite>peer_add</cite>) requires the peer cluster configuration and user keyring
to be available in the primary cluster (manager host and hosts running the mirror daemon).
This can be avoided by bootstrapping and importing a peer token. Peer bootstrap involves
creating a bootstrap token on the peer cluster via:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror peer_bootstrap create &lt;fs_name&gt; &lt;client_entity&gt; &lt;site-name&gt;
</pre></div>
</div>
<p>e.g.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror peer_bootstrap create backup_fs client.mirror_remote site-remote
{&quot;token&quot;: &quot;eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ==&quot;}
</pre></div>
</div>
<p><cite>site-name</cite> refers to a user-defined string to identify the remote filesystem. In context
of <cite>peer_add</cite> interface, <cite>site-name</cite> is the passed in <cite>cluster</cite> name from <cite>remote_cluster_spec</cite>.</p>
<p>Import the bootstrap token in the primary cluster via:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror peer_bootstrap import &lt;fs_name&gt; &lt;token&gt;
</pre></div>
</div>
<p>e.g.:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ==
</pre></div>
</div>
</div>
<div class="section" id="mirroring-status">
<h2>Mirroring Status<a class="headerlink" href="#mirroring-status" title="Permalink to this headline">¶</a></h2>
<p>CephFS mirroring module provides <cite>mirror daemon status</cite> interface to check mirror daemon status:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror daemon status
[
  {
    &quot;daemon_id&quot;: 284167,
    &quot;filesystems&quot;: [
      {
        &quot;filesystem_id&quot;: 1,
        &quot;name&quot;: &quot;a&quot;,
        &quot;directory_count&quot;: 1,
        &quot;peers&quot;: [
          {
            &quot;uuid&quot;: &quot;02117353-8cd1-44db-976b-eb20609aa160&quot;,
            &quot;remote&quot;: {
              &quot;client_name&quot;: &quot;client.mirror_remote&quot;,
              &quot;cluster_name&quot;: &quot;ceph&quot;,
              &quot;fs_name&quot;: &quot;backup_fs&quot;
            },
            &quot;stats&quot;: {
              &quot;failure_count&quot;: 1,
              &quot;recovery_count&quot;: 0
            }
          }
        ]
      }
    ]
  }
]
</pre></div>
</div>
<p>An entry per mirror daemon instance is displayed along with information such as configured
peers and basic stats. For more detailed stats, use the admin socket interface as detailed
below.</p>
<p>CephFS mirror daemons provide admin socket commands for querying mirror status. To check
available commands for mirror status use:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph --admin-daemon /path/to/mirror/daemon/admin/socket help
{
    ....
    ....
    &quot;fs mirror status cephfs@360&quot;: &quot;get filesystem mirror status&quot;,
    ....
    ....
}
</pre></div>
</div>
<p>Commands with <cite>fs mirror status</cite> prefix provide mirror status for mirror enabled
file systems. Note that <cite>cephfs&#64;360</cite> is of format <cite>filesystem-name&#64;filesystem-id</cite>.
This format is required since mirror daemons get asynchronously notified regarding
file system mirror status (A file system can be deleted and recreated with the same
name).</p>
<p>Right now, the command provides minimal information regarding mirror status:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@360
{
  &quot;rados_inst&quot;: &quot;192.168.0.5:0/1476644347&quot;,
  &quot;peers&quot;: {
      &quot;a2dc7784-e7a1-4723-b103-03ee8d8768f8&quot;: {
          &quot;remote&quot;: {
              &quot;client_name&quot;: &quot;client.mirror_remote&quot;,
              &quot;cluster_name&quot;: &quot;site-a&quot;,
              &quot;fs_name&quot;: &quot;backup_fs&quot;
          }
      }
  },
  &quot;snap_dirs&quot;: {
      &quot;dir_count&quot;: 1
  }
}
</pre></div>
</div>
<p><cite>Peers</cite> section in the command output above shows the peer information such as unique
peer-id (UUID) and specification. The peer-id is required to remove an existing peer
as mentioned in the <cite>Mirror Module and Interface</cite> section.</p>
<p>Command with <cite>fs mirror peer status</cite> prefix provide peer synchronization status. This
command is of format <cite>filesystem-name&#64;filesystem-id peer-uuid</cite>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror peer status cephfs@360 a2dc7784-e7a1-4723-b103-03ee8d8768f8
{
  &quot;/d0&quot;: {
      &quot;state&quot;: &quot;idle&quot;,
      &quot;last_synced_snap&quot;: {
          &quot;id&quot;: 120,
          &quot;name&quot;: &quot;snap1&quot;,
          &quot;sync_duration&quot;: 0.079997898999999997,
          &quot;sync_time_stamp&quot;: &quot;274900.558797s&quot;
      },
      &quot;snaps_synced&quot;: 2,
      &quot;snaps_deleted&quot;: 0,
      &quot;snaps_renamed&quot;: 0
  }
}
</pre></div>
</div>
<p>Synchronization stats such as <cite>snaps_synced</cite>, <cite>snaps_deleted</cite> and <cite>snaps_renamed</cite> are reset
on daemon restart and/or when a directory is reassigned to another mirror daemon (when
multiple mirror daemons are deployed).</p>
<p>A directory can be in one of the following states:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>- `idle`: The directory is currently not being synchronized
- `syncing`: The directory is currently being synchronized
- `failed`: The directory has hit upper limit of consecutive failures
</pre></div>
</div>
<p>When a directory hits a configured number of consecutive synchronization failures, the
mirror daemon marks it as <cite>failed</cite>. Synchronization for these directories are retried.
By default, the number of consecutive failures before a directory is marked as failed
is controlled by <cite>cephfs_mirror_max_consecutive_failures_per_directory</cite> configuration
option (default: 10) and the retry interval for failed directories is controlled via
<cite>cephfs_mirror_retry_failed_directories_interval</cite> configuration option (default: 60s).</p>
<p>E.g., adding a regular file for synchronization would result in failed status:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph fs snapshot mirror add cephfs /f0
$ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror peer status cephfs@360 a2dc7784-e7a1-4723-b103-03ee8d8768f8
{
  &quot;/d0&quot;: {
      &quot;state&quot;: &quot;idle&quot;,
      &quot;last_synced_snap&quot;: {
          &quot;id&quot;: 120,
          &quot;name&quot;: &quot;snap1&quot;,
          &quot;sync_duration&quot;: 0.079997898999999997,
          &quot;sync_time_stamp&quot;: &quot;274900.558797s&quot;
      },
      &quot;snaps_synced&quot;: 2,
      &quot;snaps_deleted&quot;: 0,
      &quot;snaps_renamed&quot;: 0
  },
  &quot;/f0&quot;: {
      &quot;state&quot;: &quot;failed&quot;,
      &quot;snaps_synced&quot;: 0,
      &quot;snaps_deleted&quot;: 0,
      &quot;snaps_renamed&quot;: 0
  }
}
</pre></div>
</div>
<p>This allows a user to add a non-existent directory for synchronization. The mirror daemon
would mark the directory as failed and retry (less frequently). When the directory comes
to existence, the mirror daemons would unmark the failed state upon successfull snapshot
synchronization.</p>
<p>When mirroring is disabled, the respective <cite>fs mirror status</cite> command for the file system
will not show up in command help.</p>
</div>
<div class="section" id="configuration-options">
<h2>Configuration Options<a class="headerlink" href="#configuration-options" title="Permalink to this headline">¶</a></h2>
<dl class="std confval">
<dt id="confval-cephfs_mirror_max_concurrent_directory_syncs">
<code class="sig-name descname"><span class="pre">cephfs_mirror_max_concurrent_directory_syncs</span></code><a class="headerlink" href="#confval-cephfs_mirror_max_concurrent_directory_syncs" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div><p>maximum number of directory snapshots that can be synchronized
concurrently by cephfs-mirror daemon. Controls the number of
synchronization threads.</p>
<dl class="field-list simple">
<dt class="field-odd">type</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">uint</span></code></p>
</dd>
<dt class="field-even">default</dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">3</span></code></p>
</dd>
<dt class="field-odd">min</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">1</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt id="confval-cephfs_mirror_action_update_interval">
<code class="sig-name descname"><span class="pre">cephfs_mirror_action_update_interval</span></code><a class="headerlink" href="#confval-cephfs_mirror_action_update_interval" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div><p>处理未完成镜像更新操作的时间间隔，单位为秒。</p>
<dl class="field-list simple">
<dt class="field-odd">type</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">secs</span></code></p>
</dd>
<dt class="field-even">default</dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">2</span></code></p>
</dd>
<dt class="field-odd">min</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">1</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt id="confval-cephfs_mirror_restart_mirror_on_blocklist_interval">
<code class="sig-name descname"><span class="pre">cephfs_mirror_restart_mirror_on_blocklist_interval</span></code><a class="headerlink" href="#confval-cephfs_mirror_restart_mirror_on_blocklist_interval" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div><p>Interval in seconds to restart blocklisted mirror instances. Setting
to zero (0) disables restarting blocklisted instances.</p>
<dl class="field-list simple">
<dt class="field-odd">type</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">secs</span></code></p>
</dd>
<dt class="field-even">default</dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">30</span></code></p>
</dd>
<dt class="field-odd">min</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">0</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt id="confval-cephfs_mirror_max_snapshot_sync_per_cycle">
<code class="sig-name descname"><span class="pre">cephfs_mirror_max_snapshot_sync_per_cycle</span></code><a class="headerlink" href="#confval-cephfs_mirror_max_snapshot_sync_per_cycle" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div><p>maximum number of snapshots to mirror when a directory is picked up
for mirroring by worker threads.</p>
<dl class="field-list simple">
<dt class="field-odd">type</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">uint</span></code></p>
</dd>
<dt class="field-even">default</dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">3</span></code></p>
</dd>
<dt class="field-odd">min</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">1</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt id="confval-cephfs_mirror_directory_scan_interval">
<code class="sig-name descname"><span class="pre">cephfs_mirror_directory_scan_interval</span></code><a class="headerlink" href="#confval-cephfs_mirror_directory_scan_interval" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div><p>interval in seconds to scan configured directories for snapshot
mirroring.</p>
<dl class="field-list simple">
<dt class="field-odd">type</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">uint</span></code></p>
</dd>
<dt class="field-even">default</dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">10</span></code></p>
</dd>
<dt class="field-odd">min</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">1</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt id="confval-cephfs_mirror_max_consecutive_failures_per_directory">
<code class="sig-name descname"><span class="pre">cephfs_mirror_max_consecutive_failures_per_directory</span></code><a class="headerlink" href="#confval-cephfs_mirror_max_consecutive_failures_per_directory" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div><p>number of consecutive snapshot synchronization failues to mark a
directory as “failed”. failed directories are retried for
synchronization less frequently.</p>
<dl class="field-list simple">
<dt class="field-odd">type</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">uint</span></code></p>
</dd>
<dt class="field-even">default</dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">10</span></code></p>
</dd>
<dt class="field-odd">min</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">0</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt id="confval-cephfs_mirror_retry_failed_directories_interval">
<code class="sig-name descname"><span class="pre">cephfs_mirror_retry_failed_directories_interval</span></code><a class="headerlink" href="#confval-cephfs_mirror_retry_failed_directories_interval" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div><p>interval in seconds to retry synchronization for failed directories.</p>
<dl class="field-list simple">
<dt class="field-odd">type</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">uint</span></code></p>
</dd>
<dt class="field-even">default</dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">60</span></code></p>
</dd>
<dt class="field-odd">min</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">1</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt id="confval-cephfs_mirror_restart_mirror_on_failure_interval">
<code class="sig-name descname"><span class="pre">cephfs_mirror_restart_mirror_on_failure_interval</span></code><a class="headerlink" href="#confval-cephfs_mirror_restart_mirror_on_failure_interval" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div><p>Interval in seconds to restart failed mirror instances. Setting to
zero (0) disables restarting failed mirror instances.</p>
<dl class="field-list simple">
<dt class="field-odd">type</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">secs</span></code></p>
</dd>
<dt class="field-even">default</dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">20</span></code></p>
</dd>
<dt class="field-odd">min</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">0</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

<dl class="std confval">
<dt id="confval-cephfs_mirror_mount_timeout">
<code class="sig-name descname"><span class="pre">cephfs_mirror_mount_timeout</span></code><a class="headerlink" href="#confval-cephfs_mirror_mount_timeout" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div><p>Timeout in seconds for mounting primary or secondary (remote) ceph
file system by the cephfs-mirror daemon. Setting this to a higher
value could result in the mirror daemon getting stalled when mounting
a file system if the cluster is not reachable. This option is used to
override the usual client_mount_timeout.</p>
<dl class="field-list simple">
<dt class="field-odd">type</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">secs</span></code></p>
</dd>
<dt class="field-even">default</dt>
<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">10</span></code></p>
</dd>
<dt class="field-odd">min</dt>
<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">0</span></code></p>
</dd>
</dl>
</div></blockquote>
</dd></dl>

</div>
<div class="section" id="re-adding-peers">
<h2>Re-adding Peers<a class="headerlink" href="#re-adding-peers" title="Permalink to this headline">¶</a></h2>
<p>When re-adding (reassigning) a peer to a file system in another cluster, ensure that
all mirror daemons have stopped synchronization to the peer. This can be checked
via <cite>fs mirror status</cite> admin socket command (the <cite>Peer UUID</cite> should not show up
in the command output). Also, it is recommended to purge synchronized directories
from the peer  before re-adding it to another file system (especially those directories
which might exist in the new primary file system). This is not required if re-adding
a peer to the same primary file system it was earlier synchronized from.</p>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../client-config-ref/" class="btn btn-neutral float-right" title="客户端配置" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../snap-schedule/" class="btn btn-neutral float-left" title="Snapshot Scheduling Module" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>